Computerized video file analysis tool and method

ABSTRACT

A computer-implemented method for dynamically creating and presenting video content information to a user of a computer having an associated screen involves: i) loading contents of a video file into a video player; ii) displaying frames of the video file; iii) receiving a user&#39;s input indicating selection of an object displayed in at least one frame; iv) performing an object identification analysis of frames to locate each instance where a specific frame contains the object; v) for each specific frame that contains the object, performing a z-axis analysis of the object to determine prominence of the object within each specific frame; vi) storing metadata indicating results of the object identification analysis and, for frames where the object was present, the z-axis analysis; and vii) automatically generating and displaying a graphical timeline display graphically reflecting frames containing the object and object prominence within those frames based upon the metadata.

FIELD OF THE INVENTION

This disclosure relates generally to computerized video analysis toolsand, more particularly, to improved computerized video analysis andsearch tools.

BACKGROUND

As the popularity of online video increases, so do the number of videoshosted on video streaming sites such as, for example, YouTube® andNetflix® to name a few. It is estimated that, on YouTube alone, over 4billion videos are viewed each day, and that 300 hours of new video areuploaded to YouTube every minute. For users seeking to find specificcontent in a part of a video, largely gone are the days of simplyrewinding and fast forwarding through it. A user seeking to reviewcontent in an online video presently has a number of search andscrubbing solutions. Certain video player software allows users to“scrub” through a video (i.e., move through the video timeline typicallyby dragging a pointer on a slider from left to right) whereby thumbnailsof key frames of the video are shown. This enables a user to quicklyscan the content of a video to see what is coming up or has gone before.FIG. 1 illustrates, in simplified form, a simplified example of aconventional, prior art video player 100, having a user interface 102,that is running on a processor-containing computer 104 (which could be asmart television, a desktop computer, a laptop computer, a tabletcomputer, a smart phone, a smart watch, or other computing devicecapable of playing video for a user in a video player). As isconventional, the computer will contain one or more processors, as wellas RAM, ROM, some form of I/O, and non-volatile storage.

As shown in FIG. 1, and as is well known, the user interface 102 of thevideo player 100 includes interface controls such as a conventionalPlay/Pause button 106, a fast forward button 108, a rewind button 110, astop button 112 and one or more auxiliary buttons, for example, a volumecontrol button 114. The user interface 102 also includes a slider 116via which the user can scrub through a video loaded into, or streamingto, the video player 100.

As shown in FIG. 1, the current video is paused at a point partiallythrough the video, as indicated by a pointer 118 of the slider 116. Thecurrent frame of the paused video is displayed within the screen 120 ofthe video player 100 and shows a human figure 122 in the center, aseries of buildings 124 a, 124 b, 124 c, and a balloon 126 floatingbetween the location of the figure 122 and one of the buildings 124 a.In addition, the screen 120 contains a timeline 128 of the currentlyloaded video that includes a series of key frames 130 a, 130 b, 130 c,130 d, 130 e, 130 f, 130 g, 130 h, 130 i that correspond to some numberof frames 130 a, 130 b, 130 c, 130 d of the video before thecurrently-displayed frame 130 e and some number of frames 130 f, 130 g,130 h, 130 i of the video after the currently-displayed frame 130 e. Inaddition, with this particular user interface 102, thecurrently-displayed frame 130 e is shown enlarged on the timeline. Ascan be seen in the subset of key frames 130 a, 130 b, 130 c, 130 d, 130e, 130 f, 130 g, 130 h, 130 i, the balloon 126 is traversing from theleft side of the frame, behind the figure 122 and in front of thebuildings 124 a, 124 b, 124 c. If the user wanted to locate where in thevideo, for example, the balloon is above the second building 124 b, theywould move the pointer 118 of the slider 116 (in this case simplyadvance it to the right) until the specific frame 130 h was located. Ofcourse, if that frame 130 h was not within the displayed portion of thetimeline and its specific location was unknown, the user might have tomove the pointer 118 back and forth along the slider 116 until theparticular frame of interest was located.

While the display of key frames 130 a-130 i can assist a user in findinga desired part of a video, this type of searching can be time consumingand tedious and presents a problem because this approach is prone tohaving the user overshoot, or entirely miss, key frames of interest.

The above problem is compounded if the searching is to be donerepeatedly for multiple videos. For example, it is presently notuncommon for old films to be digitized so that they can be made morebroadly available for various purposes, including scholarly research. Indoing so, when digitized, the videos may have some associatedinformation logged for future reference relating to its content, butthat information typically only reflects the major focus of the film andmay not include minor details that are not noteworthy at the time, or ofno interest per se. As a result, it is likewise not uncommon for a laterresearcher viewing a digitized video to notice someone, or something,previously unnoticed that is later recognized to be of significance, forexample, the presence of a person long before they were famous or adetail that may aid in unraveling some long unsolved mystery. Suchresearch efforts can require, a researcher to view countless hours ofvideos of potentially no relevance at all with the hope that they maypossibly contain a few seconds of the desired person(s) or thing(s).

Thus, there is an ongoing and increasing problem involving the abilityto more quickly and efficiently perform video searching.

SUMMARY

One aspect of this disclosure involves a computer-implemented method fordynamically creating and presenting video content information to a userof a computer having an associated screen. The method involves: i) usinga processor of the computer, loading contents of a video file into avideo player, for display in a user interface of the video player on thescreen; ii) displaying frames of the video file in the user interface;iii) receiving, via the user interface, a user's input indicatingselection of an object displayed in at least one frame of the videofile; iv) performing, using the processor, an object identificationanalysis of frames comprising the video file to locate each instancewhere a specific frame of the video file contains the object; v) foreach specific frame of the video file that contains the object,performing a z-axis analysis of the object within the frame, using theprocessor, to determine prominence of the object within each specificframe; vi) storing metadata in non-volatile storage associated with thevideo file, the metadata indicating results of the object identificationanalysis and, for frames where the object was present, the z-axisanalysis; and vii) using the processor, automatically generating anddisplaying for the video file, on the screen synchronized to the videofile, a graphical timeline display for the user graphically reflectingframes of the video file containing the object and object prominencewithin those frames based upon the metadata.

The foregoing and following outlines rather generally the features andtechnical advantages of one or more embodiments of this disclosure inorder that the following detailed description may be better understood.Additional features and advantages of this disclosure will be describedhereinafter, which may form the subject of the claims of thisapplication.

BRIEF DESCRIPTION OF THE DRAWINGS

This disclosure is further described in the detailed description thatfollows, with reference to the drawings, in which:

FIG. 1 shows, in simplified form, a simplified example of aconventional, prior art video player, having a user interface, runningon a conventional processor-containing computer device;

FIG. 2 represents, in simplified form, a portion of a cartoon that hasbeen processed using an implementation of the tool described herein;

FIG. 3 represents, in simplified form, a different portion of thecartoon discussed in connection with FIG. 2;

FIG. 4 illustrates, in simplified form, a video player similar to thevideo player of FIG. 1, except that it has been enhanced by the additionof a tool variant as described herein;

FIG. 5 illustrates, in simplified form, a representation of a portion ofa video following analysis using one tool variant constructed asdescribed herein;

FIG. 6 illustrates, in simplified form, a portion of a video followinganalysis using one tool variant constructed as described herein, withthe results of the analysis graphically presented underneath;

FIG. 7A illustrates, in simplified form, an example single timelinegenerated by an alternative variant tool, that implements Booleancombination of selections (“OR”), for the same video portion andselections of FIG. 6;

FIG. 7B illustrates, in simplified form, an example single timelinegenerated by an alternative variant tool, that implements Booleancombination of selections (“AND”), for the same video portion andselections of FIG. 6;

FIG. 7C illustrates, in simplified form, an example single timeline 710c generated by an alternative variant tool, that implements Booleancombination of selections (“XOR”), for the same video portion 600 andselections of FIG. 6;

FIG. 8 illustrates, in simplified form, another portion of a video andan associated timeline as would be generated for that portion, afterselection of a mouse, by a further variant of a tool constructed asdescribed herein; and

FIG. 9 illustrates, in simplified form, a computer running a variant ofthe tool described herein that is constructed so that it can performsearches across multiple files based upon a user selection of one ormore objects.

DETAILED DESCRIPTION

This disclosure provides a technical solution to address theaforementioned problems inherent with the present capability forsearching video content. Out technical solution improves upon currentvideo players used on computer devices by analyzing video and generatinga modified display that graphically identifies those frames of the videowhere a user-selected object appears and how prominent that object is inthe respective frames. Further refinements of our technical solutionallow for two or more objects to be selected and the modified displaywill indicate, depending upon the particular implementation and/or userselection, the frames where each appears and their prominence or someBoolean combination of those objects such that, for example, thoseframes where any of the objects appear is identified (e.g., a logical“OR” function), only those frames where all of the objects appear isidentified (e.g., a logical “AND” function), or (in the case of twoobjects selected) only those frames where one or the other but not bothappear (e.g., a logical “Exclusive OR” function).

Still further, implementations of our technical solution furtherimproves current video players used on computer devices by allowing theanalysis to occur across multiple videos selected by a user such thatthe user can select an object in a single video and the presence andprominence of that object in that video and other videos can beidentified and displayed.

By dynamically creating and displaying this information the computer'sability to convey information about user identified content in a videoand the computer's ability to interact with a user via a video player isimproved.

Specifically, our technical solution to the above problem is implementedas a tool that is either added as an extension to the user interface ofa video player, such as described above in connection with FIG. 1, or isincorporated into the implementation of a video player, and, in eithercase, operates to provide for better interaction between a computer anduser with respect to video by, upon selection of an object or objectsdisplayed somewhere in the video, automatically analyzing the frames ofthe to identify whether, and where, the particular specified object(s)(e.g., person(s) or thing(s)) are present in a video, and theirprominence where present, and automatically dynamically generating atimeline display for the video containing the results of the analysis,without the user having to view the entire video.

The tool will typically be implemented in software using program codethat is compatible with the particular video player with which it willbe used, although for specialized devices, aspects of the tool and itsfunction can be implemented in hardware. In addition, depending upon theparticular implementation, the tool can be programmed to take advantageof specialized processing and/or rendering capabilities that may beprovided by a specific graphic processing unit (GPU) that may beassociated with, or contained within, the computer that will be runningthe video player itself.

Some implementations of our tool can further significantly enhance andtransform the process of video search by operating across multiplevideos in response to a user selection of particular object(s) in onevideo and providing a timeline display of whether, and where, anyparticular object(s) (e.g., person(s) or thing(s)) are present in eachwithout having to individually view each.

As an initial matter, it is to be noted that digital video is currentlyproduced in any number of formats and, in some cases, embedded in aparticular format container. It is to be understood that the tool andoperation described herein is intended to be applicable for use withdigital video files having, for example, one of the currently used fileextension such as: .aaf, .3gp, .asf, .wmv, .avi, .flv, .swf, .mkv, .mov,.mpeg, .mpg, .mpe, .mp4, .mxf, .nsv, .ogg, .rm, to name a few, as wellas any other video file formats and containers that may be developed orbecome prevalent in the future.

In this application, for clarity, certain terms are to be understood tohave the following definitions.

The term “frame” is to be understood to mean and encompass any timeinstance of digital video that can be viewed in a video player, withoutregard to actual correspondence to a conventional “frame” as that termwould be understood for physical film or to a “cel” or “key frame” oftraditional animation. Thus, interpolated video between two key framescan constitute a “frame” as defined and referred to herein.

The term “object” when used herein in connection with a video isintended to mean and encompass anything depicted or appearing within aframe of a video including, but not limited to, a person or a thing,whether or not it exists, or can exist, in the real world. For example,real people, structures and items are intended to be “objects” asdefined herein, as is anything appearing in any type of cartoon or other“drawn” or animated image (e.g. characters, vehicles, items, speech orthought bubbles, lightning bolts, representations or manifestations ofcharacter powers or phenomena, etc.).

Also, the term “z-axis” is intended to indicate and mean an imaginarydirection into the plane of the screen of the video player, a “z-axisposition” is intended to mean a perceived depth of an object within thevideo, if what is displayed actually existed and was viewed in 3dimensional space, and “prominence” is intended to mean an indication ofthe perceived position of the object within the video relative to theplane of the screen, with greater “prominence” being indicative of beingcloser to the plane of the screen and lesser “prominence” beingindicative of farther distance from the plane of the screen.

Finally, it is to be understood that the instant tool is intended to beagnostic as to the presence or absence of any audio that may beassociated with or synchronized to the video.

Now, by way of general operational overview, we extend the capability ofa conventional video player so that a user can select an objectappearing within a frame of a video, for example, by clicking/tapping onit in the video player, or by typing text specifying the object in adesignated field. Upon doing so, in the case of text entry, the toolwill correlate the text entry with an object in the video, using imageidentification software, to identify the selected object. The tool willthen analyze the video to determine the video frames where the objectoccurs, and where on the z-axis the object is placed using objectoccurrence analysis and z-axis analysis. Then, a graphic will berendered so as to show the frames in which the object is visible andconcurrently represent its z-axis position within such frames. Moreover,as noted above, some implementations of the tool extend that capabilityfurther to allow a user to specify two or more videos and by selectingan object in one, the tool will automatically search that video and theother selected videos and generate and display a graphic indicating thepresence and prominence of that object in all of the selected videos.With such implementations, a user need not view every video, or theentirety thereof, but can merely review some portion(s) of the specifiedvideos that the tool identifies as containing the desired object(s).Moreover, at all times, the user can limit their review, if desired, toa further subset of the identified section(s) in which the objectappears in a more prominent position within the video(s).

By way of simplified example, FIG. 2 represents, in simplified form, aportion 200 of a cartoon that has been processed using an implementationof the tool described herein. As shown, the portion 200 is made up offour frames 202 a, 202 b, 202 c, 202 d. The video has been analyzed as aresult of a user's selection of the head 204 of the broom 206 in thefirst shown frame 202 a, as indicated by the selection box 208. Theresults of that analysis is contained in the auxiliary timeline 210presented, in this example, under the portion 200. As shown, thepresence of the object, in this case the head 204 of the broom 206, isindicated by lines 212 a, 212 b, 212 c in the frames 202 a, 202 b, 202 cin which it appears. In addition, any frame 202 d where the object doesnot appear is indicated, for example, by a cross hatched or grey area214 in the auxiliary timeline 210 for that frame 202 d.

More particularly, in this representation, the closer a line is to thebottom of the timeline 210, the more prominent (i.e., “forward”) in theframe the object is. As such, it can be seen that, in the first frame202 a, the head 204 of the broom 206 is just over halfway into the“depth” of the frame 202 a and so the line 212 a in the timeline 210indicating its presence is just over halfway to the top. In the nextframe 202 b, the head 204 of the broom 206 has moved forward a bit toabout halfway into the “depth” of the frame 202 b and so the line 212 bin the timeline 210 indicating its presence is now about midway betweenthe top and bottom of the timeline 210. In the third displayed frame 202c, the head 204 of the broom 206 is now substantially further forward,so the line 212 c in the timeline 210 indicating its presence is shownnearly to the bottom of the timeline 210.

In a similar vein, FIG. 3 represents, in simplified form, a differentportion 300 of the cartoon discussed in connection with FIG. 2. Asshown, the user has selected the face 304 of the tall lanky character.Through its object occurrence analysis, the tool has identified allframes where that character's face 304 appears, which includes fourframes 302 b, 302 c, 302 e, 302 f of the six frames 302 a, 302 b, 302 c,302 d, 302 e, 302 f of the cartoon shown in FIG. 3. Through z-axisanalysis, the tool has also determined the prominence of that face 304in each frame of the cartoon it is present and indicated it with lines306, 308, 310, 312 in the timeline 210. As with FIG. 2, in FIG. 3, across hatched or grey area 214 is shown in the auxiliary timeline 210for the range of frames where the face 304 does not appear.

With the foregoing general understanding, the details of variousimplementations for various parts of the tool will now be discussed.

Object Selection

Depending upon the particular implementation, a user selects an objectto locate and track within a video using one of two methods, visualselection or textual search.

With visual selection, a user selects an object as it appears in avideo, via the user interface, by selecting an object directly, forexample, by clicking or tapping on it or by surrounding it using someform of selection tool, like a box, oval, freeform selection tool, etc.

With textual search, a user types the “name” of an object to track, forexample, “ball,” “hammer,” “mouse,” or “Mr. Kat.” The tool will thencheck stored prior tool-generated metadata associated with that video todetermine whether that “name” has previously been searched and, if so,the object to which it pertains. If that “name” has already previouslybeen searched, in different implementations, different things canhappen.

In some cases, the prior search will have previously caused the tool tolocate the object and its prominence in the video and will havegenerated and stored metadata associated for that object “name”synchronized to the video such that it can merely be retrieved togenerate and display the timeline for the user relating to that objectand its prominence.

In cases where the “name” has not previously been searched, the toolwill do one or more of the following, depending upon the specificimplementation variant.

For some variants, it will require the user to go to some part of thevideo where the desired object appears, provide a descriptive “name” forthe object, and, using a graphical selection tool identify the boundaryof the desired object. From that point, the textual selection istransformed into an object selection and proceeds in that manner, butincludes the specified “name” in the stored metadata for future use.

For other variants, where the user's computer has internet access, thetool can access the image search capability of a search engine, forexample, the Google® search engine, and will conduct an image search onthe internet corresponding to the “name” and use results of the imagesearch to locate an object in the video frames that corresponds to theimage that was returned by the search. Again, this may initially entailpresenting image search results and asking the user to select one ormore so that the tool can “learn” a correspondence between the “name”and images. For example, if the user typed the name of a cartooncharacter Mr. Kat, the tool might display a window containing thumbnailimages of some of the image search results, which could include multipleimages of, for example, actual cats, the cartoon characters Krazy Kat,Garfield the cat, and the desired “Mr. Kat” and ask the user to selectone or more corresponding to the name “Mr. Kat.” The tool would then usethe selected image(s) as the object for the object search.

Irrespective of the manner in which an object is selected by the user,the selection triggers an automatic process that is functionally made upof two parts: (1) object detection (also called object search) analysisand (2) z-axis analysis.

Object Detection/Search Analysis

Once an object has been selected by a user, the tool searches the videofor each frame where that object appears. Numerous algorithms andapproaches for identification of an object in a video and tracking theobject within the video (i.e. following its movement from frame toframe) presently exist, particularly for use in security applications,and thus can be straightforwardly used, or adapted for use, in a tool asdescribed herein. Some representative examples include, but are notlimited to, object identification and tracking techniques disclosed inU.S. Pat. Nos. 7,898,576, 7,391907, 8,218819, 8,559,670 and U.S. Pat.Pub. No. 2014/0328512, all of which are incorporated herein by referencein their entirety. Advantageously, such object identification andtracking techniques are useful because, in some cases, they can accountfor variations in the object due to, for example, rotation of the objector partial obscuring, and thus continue to track it, thereby improvingthe accuracy of the identified frames containing a selected object.

Our tool augments those known object identification and trackingtechniques by maintaining a log of each frame where the object appears.Depending upon the particular implementation, the log can be created innon-volatile storage and then updated as the analysis progresses, it canbe created in temporary storage until some specified part of theanalysis is complete (which may be part or all of the object detectionor search analysis, or may include some or all of the z-axis analysis aswell) and then the log stored in the non-volatile storage. In any event,the information in the log resulting from the object detection or searchanalysis is then used as part of the z-axis analysis.

Z-Axis Analysis

Once the tool identifies each video frame that contains the object,those frames are further analyzed to determine the object's position onthe z-axis (how close to the foreground or background it is). To dothis, the tool compares the relative size of the selected object as itappears in the frame to other objects in the frame and the size of theobject across all the frames in which it appears with the larger theobject appears in a video frame being presumed to establish a positionfor the object “closer” to the front of the screen and the smaller theobject is being presumed to establish a position for the object “fartherinto” the scene displayed on the screen Likewise, the analysis may takeinto account changes in the placement of the bottom of the selectedobject relative to the bottom of the frame in conjunction with a changein the object's size as an indication of movement to or away from thefront of the screen under the presumption that what is shown is always aperspective view.

The result of the z-axis analysis adds an indication in the log for eachframe that can be used to graphically represent the prominence of theobject.

At this point, it is to be understood that the result of this analysisis not to be taken as a specific indication of a distance for the objectfrom the screen or an imputed distance if translated to the real world.Rather, at best, it is merely intended as an indication of relativeprominence of placement within the “world” shown in that particularvideo and, in some cases, it may only be valid within a given “scene” ofthe video. For example, placement of an object within a room may beindicated as having the same z-axis location as the same object in ascene showing the object on a street, even though, if translated to thereal world, the object would have very different actual distances fromthe plane of the “screen” in the two scenes.

Following the z-axis analysis, the identification of the frames in whichthe selected object appears and its prominence will have beendetermined, so he log file is updated and, if not previously stored, itis stored in non-volatile storage associated with the video. Dependingupon the particular implementation of the tool and the type of file orits container, this may involve modifying the metadata file alreadyassociated with the video (i.e., creating a new “enhanced” metadatafile) or creating an entirely new file that is associated with the videofile that can be accessed by the tool.

Graphic Rendering of Results

At a point after the foregoing analysis is complete, which may beimmediately thereafter or, if the current object selection correspondsto a previous object selection, at some point thereafter, the tool willaccess the information resulting from the analysis and use it to rendera graphic, synchronized to the video that indicates each frame in whichthe selected object appeared and its prominence in those individualframes, and display that graphic for the user.

Depending upon the particular implementation, the rendering can bedisplayed in some portion of the video player screen, as an overlay ontop of some portion of the video screen, or in an entirely separatearea, for example, a separate window on the computer screen, theimportant aspect being what is shown, not where the graphic is shown.

Typically, the graphic rendering of the results will be presented in atimeline form, which may concurrently show the object-relatedinformation for the entire video or, if the video is too long, only aportion may be viewable at any given time. Alternatively, the tool mayinclude the capability to expand or compress the timeline so that agreater or lesser number of frames is encompassed within it.

In any event, the graphic rendering is configured such that the user canrecognize those frames where the selected object appears. This may bedone in any suitable manner including, for example, using lines, dots,different thickness or patters of each, colors, color shades, etc., theimportant aspect being that an intelligible visual indication ispresented to the user, not the manner in which the information isgraphically presented or conveyed.

Depending upon the particular implementation and video player/toolcombination, the user can then be presented with any one or more ofseveral reviewing abilities. For example, with some implementations, theuser can be given the ability to “play” the video after object selectionand only those frames containing the selected object will be played(i.e., the player will jump past any frames that do not contain theobject). With other implementations, the user will be able to use aslider (of the video player or separately provided by the tool) to scrubthrough the frames containing the selected object. With still otherimplementations, the user will have the ability to select a particularpoint in the graphic, as presented, which will enable them to goimmediately to a particular frame within the portion of the videocontaining the object.

FIG. 4 illustrates, in simplified form, a video player 400, similar tothe video player 100 of FIG. 1, except that it has been enhanced by theaddition of a tool variant as described herein.

As shown, the video from FIG. 1 has previously been loaded and is pausedsuch that the same frame from FIG. 1 is shown on the screen 420 of thevideo player 400. A user has selected the balloon 126 in that frameusing a selection tool 402. As a result of the selection, the toolanalyzed the video for the presence and prominence of the balloon 126 inthe frames of the video and the results of that analysis have beenrendered and are displayed in a timeline 210 contained within the screen420 of the video player. Note that, in this particular exampleimplementation, the timeline 210 replaces the timeline 128 of FIG. 1,although it could have just as readily been presented above thattimeline 128 or in some other manner. As shown, the ranges of frameswhere the balloon 126 does not appear shown by cross hatching 422. Inaddition, the present frame is indicated in the timeline 210 by a dot404 located on one 414 of the lines 406, 408, 410, 412, 414, 416indicating the z-axis prominence of the balloon 126 in the frames of thevideo. As shown, with this timeline, the closer the line is to thebottom 418 of the timeline 210, the closer to the “screen” the object,in this case the balloon 126, is. Of course, this convention and theidentification indicator is arbitrary. Other implementations could usethe bottom 418 of the timeline 210 as representing the farthest distancefrom the “screen” an object can be, and still others could use someother visual indication.

In addition, with this implementation, the slider 118 and the dot 404are linked so that moving the slider 118 back and forth will scroll backand forth through the video will cause the dot 404 to correspondinglymove. If the slider is moved to a frame where the selected object doesnot appear, depending upon the particular implementation, it can causethe screen 420 to go blank or the slider can jump ahead to a positioncorresponding to the next frame where the selected object appears.

As previously noted above, the use of the type of object identificationand tracking algorithms allows the tool to take into accountmanipulations of the object (e.g., rotation, translation, partialobscuring, etc.) among frames. In this regard, FIG. 5 illustrates, insimplified form, a representation of a portion 500 of a video followinganalysis using one tool variant constructed as described herein, withthe results of that analysis depicted in the timeline 210 spanning thatrange of frames. As shown, a user previously paused in one of the frames502 and entered the text “musical note” in the user interface of thetool. This would have caused the tool to conduct an internet search ofthat phrase and, by image matching the search results with the framecontents, identify the object surrounded by the selection indicator 402.Based upon that identification, the presence and prominence of theselected object has been tracked and indicated for each frame by linesin the timeline 210, despite the note having been partially obscured,moved around over the course of the sequence 500.

Other Variants

Depending upon the particular implementation, further optionalenhancements to the tool can be provided.

For example, some implementation variants can allow a user to specifymore than one object. As such, those implementations can be configuredto generate and present multiple timelines, one for each objectselected. An example of this is shown in FIG. 6, which illustrates, insimplified form, a portion 600 of a video following analysis using onetool variant constructed as described herein, with the results of theanalysis graphically presented underneath. As shown, the tall character602 and the short character 604 have both been previously selected. As aresult, the tool of this variant has generated two discrete timelines210 a, 210 b. The upper timeline 210 a indicates the presence andprominence of the tall character 602, in this case with a solid line606, and the lower timeline 210 b indicates the presence and prominenceof the short character 604 by a dashed-dotted line 608. Alternatively,in other implementations, the lines in both timelines 210 a, 210 b couldhave had the same pattern and, in still other implementations, differentcolor could have been used for the lines instead of, or along with,different patterns of lines for each object.

Other implementation variants supporting multiple object selection canbe configured to allow for Boolean combinations involving the selectedobjects. Representative examples of this are shown in FIGS. 7A-7C.

FIG. 7A illustrates, in simplified form, an example single timeline 710a generated by an alternative variant tool, that implements Booleancombination of selections, for the same video portion 600 and selectionsof FIG. 6. As shown, the timeline 710 a contains lines 606, 608indicating the result of applying a Boolean logical “OR” function to theselections.

In similar manner, FIG. 7B illustrates, in simplified form, an examplesingle timeline 710 b generated by an alternative variant tool, thatimplements Boolean combination of selections, for the same video portion600 and selections of FIG. 6. As shown, the timeline 710 b containslines 606, 608 indicating the result of applying a Boolean logical “AND”function to the selections.

Likewise, FIG. 7C illustrates, in simplified form, an example singletimeline 710 c generated by an alternative variant tool, that implementsBoolean combination of selections, for the same video portion 600 andselections of FIG. 6. As shown, the timeline 710 c contains only a line606 as a result of applying a Boolean logical “XOR” function to theselections.

Now, it is to be appreciated that, in some cases, multiple instances ofa selected object can appear in a single frame. With some variants, onlythe selected object is tracked by the tool and its presence andprominence may be shown in the generated graphic, irrespective ofwhether other instances of that same object may appear in the sameframe(s). With other variants however, the tool may be configured todiscern the presence of multiple instances of the same object andindicate, for each such given frame, each instance of the object andtheir prominence within that frame. In still other variants, where suchinstances may merge such that it is not easily possible to separatelydiscern each, a different indication can be provided, for example, achange in thickness, color or pattern to indicate that at least twoobjects are present and have the same prominence or individualprominences that are too close to separately identify.

FIG. 8 illustrates, in simplified form, another portion 800 of a videoand an associated timeline 810 as would be generated for that portion800, after selection of a mouse 802, by a further variant of a toolconstructed as described herein. In the first frame 804 shown, there isa single mouse 802 in the foreground, so that existence and prominenceis indicated by a single line 806. In the next frame 808 however, anadditional mouse 802 a is present and has a different prominence. Assuch, a second line 806 a is also displayed for that frame reflectingthe presence and prominence of that mouse 802 a. In the next frame 814,the two mice 802, 802 a are so close together and a third mouse 802 b isadditionally present. As a result, the presence of the group of mice802, 802 a, 802 b is indicated by a single thicker line 806 b. Thisthicker line presence indication 806 b continues in the next frame 816but moves to reflect the change in prominence of the group of mice 802,802 a, 802 b. In the final frame 818 oft he portion 800, only a singlemouse 802 a is now present, so the presence indication 806 a returns tothe normal way of representing the prominence of a single object.

Multi-File Search Variants

As noted above, some variant implementations can be constructed suchthat, in addition to searching and generating a graphical display for aselected object in a video file, the search and generation can beconducted across multiple files.

FIG. 9 illustrates, in simplified form, a computer 104 running a variantof the tool described herein that is constructed so that it can performsearches across multiple files based upon a user selection of one ormore objects.

The computer 104 has a video player 400 running on it that incorporates,or has an associated extension or add on that adds, the toolfunctionality described herein. As shown, the video player 400 includesinterface controls 902, for example corresponding to those discussed inconnection with FIG. 1 and/or FIG. 4. As shown, a video has been loadedinto the video player 400 and been paused on a particular frame 904. Asindicated by a selection box 906, a user has selected an object 908shown in that frame for search. In addition, the user has indicated adesire for multi-file searching, for example, by for example, selectinga menu item, check box, radio button, or any other appropriate means. Asa result, the tool causes the processor to access the non-volatilestorage 910 to identify the video files contained therein. As shown, thestorage 910 contains many video files, for example, a first video file912, labeled in FIG. 9 “VF_(A)+m” to indicate that this video file haspreviously been searched, with the “+m” indicating that the resultingmetadata is part of the video file or its container, a second video file914, labeled in FIG. 9 “VF_(B)” and having an associated separatemetadata file 916, labeled in FIG. 9 “mVF_(B)”, a third video file 918,labeled in FIG. 9 “VF_(C)” as well as multiple other video filesculminating in the video file 920, labeled in FIG. 9 “VF_((Last))”, andwhich represents the universe of available searchable video files. Withthe simplest variant, that universe would automatically be searched.With other optional variants, the user can be given some control overspecifying particular files to be searched and other files to beignored.

In either case, those files are accessed and, in this representativeoptional example variant, the user is presented with a window 922identifying the available video files 924 a, 924 b, 924 c, 924 d, 924 e,924 f, . . . , 924 g that can be included in the search. The userspecifies which files to search by any provided appropriate methodwhich, as shown, involves use of a check box. The user selects thespecific files, which as shown, has involved the user selecting thecheck boxes 926 of at least the files named “VF2.mov,” “VF3.mp4,”“VF5.mpg,” “VFn.swf” as indicated.

Depending upon the particular implementation, with some implementations(and possibly depending upon the capability of the particular computer104), the tool will, for example, sequentially conduct the objectidentification search and z-axis analysis for each of the selectedfiles, whereas, with other implementations, additional instances of thetool may be launched and rune in the background to conduct the objectidentification search and z-axis analysis for each of the selectedfiles.

Upon completion of that analysis, the results of the search can bepresented. Depending upon the particular implementation variant, thismay involve some form of presentation for all of the selected files ormay involve presentation for only those files where the selected objectappears.

For purposes of example, with this example variant, only filescontaining the selected object(s) get presented. As such, the tool willcause the processor(s) to generate and display, in an appropriateinterface 928, timelines 210 x, 210 y, 210 z (indicating the locationand prominence of the selected object as previously described)associated with some form of video file identifier 930 x, 930 y, 930 zfor each file that contained the selected object 908.

Thus, with this example and as a result of the multi file search,although the user selected at least four files to be included in thesearch for the specified object 908, exactly three files were found toalso contain the selected object 908. Thus, the user could limit theirreview tot hose additional three files, whereas otherwise they wouldhave had to potentially review at least the seven files identified inthe selection window 922. Moreover, since the object identificationsearch and z-axis analysis has already been conducted and the resultinginformation stored in the non-volatile storage associated with thosethree files, they can advantageously each be brought into the videoplayer 400 and reviewed with respect to the selected object 908 withouthaving to re-generate their timelines 210 x, 210 y, 210 z.

Finally, it is worth noting that certain optional variants can bestraightforwardly extended to be applicable to multi-file searching aswell, for example, the Boolean search capability.

Finally, as a general matter, the aforementioned computer programinstructions implementing the tool are to be understood as being storedin a non-volatile computer-readable medium that can be accessed by aprocessor of the computer to cause the computer to function in aparticular manner. The non-volatile computer-readable medium may be, forexample (but not limited to), an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor apparatus, device, orpropagation medium. More specific examples (a non-exhaustive list) ofsuch computer-readable media include, but are not limited to, a portablecomputer diskette, a hard disk, a solid state disk, a random accessmemory, a read-only memory, an erasable programmable read-only memory(e.g., EPROM or Flash memory), an optical storage device, or a magneticstorage device. The computer usable program code may be transmitted toor accessed by the processor of the computer using any appropriatecommunication path including (but not limited to) wireless, wire line,optical fiber, etc.

Having described and illustrated the principles of this application byreference to one or more example embodiments, it should be apparent thatthe embodiment(s) may be modified in arrangement and detail withoutdeparting from the principles disclosed herein and that it is intendedthat the application be construed as including all such modificationsand variations insofar as they come within the spirit and scope of thesubject matter disclosed.

What is claimed is:
 1. A computer-implemented method for dynamicallycreating and presenting video content information to a user of acomputer having an associated screen, the method comprising: i) using aprocessor of the computer, loading contents of a video file into a videoplayer, for display in a user interface of the video player on thescreen; ii) displaying frames of the video file in the user interface;iii) receiving, via the user interface, a user's input indicatingselection of an object displayed in at least one frame of the videofile; iv) performing, using the processor, an object identificationanalysis of frames comprising the video file to locate each instancewhere a specific frame of the video file contains the object; v) foreach specific frame of the video file that contains the object,performing a z-axis analysis of the object within the frame, using theprocessor, to determine prominence of the object within each specificframe; vi) storing metadata in non-volatile storage associated with thevideo file, the metadata indicating results of the object identificationanalysis and, for frames where the object was present, the z-axisanalysis; and vii) using the processor, automatically generating anddisplaying for the video file, on the screen synchronized to the videofile, a graphical timeline display for the user graphically reflectingframes of the video file containing the object and object prominencewithin those frames based upon the metadata.
 2. The computer-implementedmethod of claim 1, further comprising: receiving, via the userinterface, a user's selection of an additional object displayed in atleast one frame of the video file; performing, using the processor, theobject identification analysis of frames comprising the video file tolocate each instance where specific frames of the video file contain theadditional object; for each specific frame of the video file thatcontains the additional object, performing a z-axis analysis of theadditional object within the frame, using the processor, to determineprominence of the additional object within each specific frame; storingadditional metadata in non-volatile storage associated with the videofile, the additional metadata indicating results of the objectidentification analysis pertaining to the additional object and, forframes where the additional object was present, the z-axis analysis; andusing the processor, automatically generating and displaying graphicsreflecting frames of the video file containing the additional object andadditional object prominence within those frames based upon theadditional metadata.
 3. The computer-implemented method of claim 2,further comprising: performing a user-specified Boolean combination ofthe metadata and additional metadata, and wherein the graphical timelinedisplay reflects a result of the Boolean combination.
 4. Thecomputer-implemented method of claim 1, wherein object prominence withinan individual frame is indicated by a line.
 5. The computer-implementedmethod of claim 4, wherein object prominence within the individual frameis reflected by proximity of the line to an edge of a part of thetimeline corresponding to the frame.
 6. The computer-implemented methodof claim 1, wherein the user interface includes a slider, the methodfurther comprising: synchronizing movement of the slider to movement ofa prominence indicator within the graphical timeline display.
 7. Thecomputer-implemented method of claim 1, further comprising: indicatingin the graphical timeline display a set of frames that do not containthe object.
 8. The computer-implemented method of claim 1, wherein thereceiving, via the user interface, the user's selection of the objectcomprises: receiving the user's selection of at least two objects; andwherein the graphical timeline display graphically reflects particularframes of the video file containing the at least two objects and theirrespective prominence within the particular frames.
 9. Thecomputer-implemented method of claim 8, wherein the graphical timelinedisplay comprises a separate timeline for each of the at least twoobjects.
 10. The computer implemented method of claim 1, furthercomprising: receiving an indication from the user that a multi-filesearch for the is to be performed; and performing “iv)” through “vi)”for a set of video files.
 11. The computer implemented method of claim10, further comprising: presenting an interface to the user via whichthe user can specify video files that are to be the set of video files.12. The computer implemented method of claim 10, wherein the set ofvideo files corresponds to all video files stored in non-volatilestorage accessible and displayable by the video player.
 13. The computerimplemented method of claim 10, further comprising: generating anddisplaying, for each file in the set containing the object, a timelineidentifying presence and prominence of the object in each file oft heset where the object appears.
 14. The computer implemented method ofclaim 1, wherein the user's input comprises: specifying the object byperforming a graphical selection action within the user interface. 15.The computer implemented method of claim 1, wherein the user's inputcomprises: entering a textual input indicative of the object.
 16. Thecomputer implemented method of claim 15 further comprising: performingan image search based upon the textual input; and comparing results ofthe image search with contents displayed in the at least one frame ofthe video file to identify an intended object specified by the textualinput.
 17. A computer-implemented tool, for dynamically creating andpresenting video content information to a user of a computer having anassociated screen, the tool being constructed to interoperate with avideo player, the tool comprising: an interface via which a user canprovide input indicating selection of an object displayed in at leastone frame of a video file displayed by the video player; an objectidentification analysis algorithm, executable by a processor associatedwith the tool; a z-axis analysis algorithm, executable by the processor,that determines prominence of the object within each frame identified bythe object identification analysis algorithm as containing an instanceof the object; metadata, stored in non-volatile storage associated withthe video file, the metadata reflecting a result of execution of theobject identification analysis algorithm and z-axis analysis algorithmon the video file based upon the selected object; and a user-visiblegraphical timeline display, dynamically generated based upon themetadata, that will graphically reflect frames of the video filecontaining the object and prominence of the object within those frames.18. The computer-implemented tool of claim 17 further comprising: amulti-file selection interface via which the user can specify othervideo files that are to be searched for presence and prominence of theobject.
 19. The computer-implemented tool of claim 18 furthercomprising: a generated display including, for each of the userspecified other files that are a group containing the object, afile-specific timeline reflecting prominence of the object within framesof the files of the group.